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INTRODUCTION 

Local Control Accountability Plans (LCAPs) 
represent a promising shift in accountability 
policy by encouraging districts to develop 
strategic plans designed to meet local needs. 
Inherent to the success of LCAPs is the use 
of multiple measures to describe a district's 
theory of action and to assess the effectiveness 
of subsequent implementation efforts. In this 
policy brief, we review promising practices 
from California districts and insights from 
research on multiple measures to provide 
recommendations that improve how districts 
generate, present, and use data in their LCAPs. 
We recommend that districts 

1. use multiple measures to develop 
greater coherence between inputs, 
processes, and outcomes linked to 
specific LCAP goals; 

2. employ the matrix approach to 
monitor progress over time and as 
a communication tool for internal 
stakeholders; and 

3. create infographics and narrative 
descriptions as a means to communicate 
critical information to external 
stakeholders. 

This policy brief is the second in a series 
aiming to support the development and 
implementation of effective LCAPs aligned 
with college and career readiness as a district's 
North Star. 1 


DEFINING MULTIPLE 
MEASURES 

This policy brief is focused on accountability 
systems that use multiple measures of 
different constructs (school climate, student 
engagement, English proficiency, etc.) to assess 
and report on school quality. Using multiple 
measures is not new; nearly every state issues 
school report cards with numerous data 
points. However, most of these systems hold 
districts and schools accountable — meaning 
they judge school quality and attach stakes 
to punish and reward for quality — exclusively 
through student outcome measures, most 
notably standardized tests of basic literacy 
and numeracy, and in some recent cases, 
the SAT and ACT. 2 California's revised 
accountability system is unique as it holds 
districts and schools accountable to providing 
adequate inputs (e.g., school facilities) and 
sound processes (e.g., academic standards 
implementation) in addition to improving 
student achievement on traditional outcome 
measures (e.g., standardized test scores). 

This new accountability model creates a vast 
amount of information for districts to organize, 
manage, and present both internally and 
externally to stakeholders. Additionally, LCAPs 
provide space for districts to select a diverse 
set of outcome measures, beyond standardized 
test scores. (See Figure 1 .) 
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A. CONDITIONS OF LEARNING: 


Basic: degree to which teachers are appropriately assigned and fully credentialed in the 
subject areas and for the pupils they are teaching; pupils have access to standards-aligned 
instructional materials and school facilities are maintained in good repair. (Priority 1) 

Implementation of State Standards: implementation of academic content and performance 
standards adopted by the State Board for all pupils, including English learners. (Priority 2) 

Course access: pupil enrollment in a broad course of study that includes all subject areas 
schools are required to offer. (Priority 7) 

B. PUPIL OUTCOMES: 

Pupil achievement: performance on standardized tests, score on Academic Performance 
Index, share of pupils that are college and career ready, share of English learners that become 
English proficient, English learner reclassification rate, share of pupils that pass Advanced 
Placement exams with 3 or higher, share of pupils determined prepared for college by the 
Early Assessment Program. (Priority 4) 

Other pupil outcomes: pupil outcomes in the subject areas schools are required to offer. 
(Priority 8) 

C. ENGAGEMENT: 

Parent involvement: efforts to seek parent input in decision making, promotion of parent 
participation in programs for unduplicated pupils and special need subgroups. (Priority 3) 

Pupil engagement: school attendance rates, chronic absenteeism rates, middle school drop- 
out rates, high school dropout rates, high school graduations rates. (Priority 5) 

School climate: pupil suspension rates, pupil expulsion rates, other local measures includ- 
ing surveys of pupils, parents and teachers on the sense of safety and school connectedness. 
(Priority 6) 


Figure 7 . Local Control Accountability Plan state priorities and associated measures. 



epic 


From Accountability to Actionability 


THE NEED FOR 
MULTIPLE MEASURES 


The use of multiple measures in accountability 
systems arose from a near-consensus among 
stakeholders that no single indicator can 
describe school quality sufficiently or produce 
the actionable information necessary for 
improving student outcomes. This consensus 
emerged in response to the unintended 
consequences of the No Child Left Behind 
Act of 2001 (NCLB), which used standardized 
test scores in reading and mathematics to 
make high-stakes decisions about perceived 
school quality. In particular, the singular focus 
on proficiency in literacy and numeracy has 
minimized all other student outcomes and led 
to curriculum narrowing in schools. 3 

It is important to note the differences between 
LCAPs and the Academic Performance Index 
(API), which is currently suspended and will 
not be reinstated in its previous form. The API, 
used from 1 999-201 3 to rate and rank order 
schools, is a score between 200 and 1 000 
that is calculated from an index of outcome 
measures, specifically standardized test 
scores in English-language arts, mathematics, 
history/social science, and science. 4 The eight 
state priorities that districts are required to 
address in LCAPs through multiple measures 
acknowledges the limitations of NCLB-era 
accountability and recognizes that assessing 
school quality goes beyond outcomes to 
include inputs and process measures. However, 
districts and schools will not immediately be 
subject to single score ratings such as the API, 
in part because the state has no measurement 


system in place for certain LCAP measures 
and because the research base demonstrating 
validity and reliability for these measures is 
evolving. 

The focus of accountability in California has 
shifted from improving standardized test 
scores to improving performance relative to 
23 additional measures. This shift requires 
districts to create an entirely new system that 
effectively addresses all 24 measures (see 
Figure 1) while also providing stakeholders 
with useful information. Instead of cataloging 
and describing every characteristic of effective 
multiple measure systems — and there are 
many — we will limit our focus to actionable 
data. Systems that promote actionability go 
beyond accountability, creating the conditions 
necessary to monitor inputs, processes, and 
outcomes. Creating these conditions allows 
districts and key stakeholders to have a 
clear sense of how the various parts of the 
educational system fit together and how their 
work relates to the district's overall goals. 
Districts also need a sense of how they are 
doing, relative to past performance and future 
goals. Detailed performance data, combined 
with a clear theory of action for improving 
student outcomes, will allow districts and their 
stakeholders to work together to diagnose 
problems early and prescribe solutions for 
immediate action. 
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RECOMMENDATIONS 


Our first recommendation is designed to 
help districts improve internal coherence by 
aligning input, process, and outcome measures 
in LCAPs. Our second recommendation 
presents an approach for translating LCAP 
information so all stakeholders can view past 
and current performance, as well as future 
expectations. Finally, we describe the methods 
some districts have used to tell their stories 
and summarize LCAPs through infographics 
and narrative descriptions. 

The recommendations are illustrated with 
promising practices from districts that 
participated in the Orange County Department 
of Education College and Career Readiness 
Consortium. The consortium focused on 
aligning LCAPs to college and career readiness 
as well as developing plans that use multiple 
measures to promote internal coherence. 

The examples in this policy brief are based on 
districts' revised LCAPs for 201 5-1 6. 


RECOMMENDATION #1: Align Input, 
Process, and Outcome Measures 

Entitled "Basic," the first state priority 
establishes a clear foundation for success. 
Students who attend schools with poor 
facilities, are taught by unqualified teachers, 5 
and lack access to quality textbooks, 
instructional materials, and technology 6 
are less likely to succeed academically 
when compared to students in schools with 
adequate inputs. Measures of the educational 
process such as access to (priority 7) and 


implementation of rigorous and rich curricula 
(priority 2), parental involvement (priority 
3), school climate (priority 6), and student 
engagement (priority 5) share importance 
because processes build off inputs to generate 
student outcomes. Systematic approaches 
to school improvement require districts to 
decipher how inputs, processes, and outcomes 
can align (see Figure 2). 


The conditions under which 
education takes place 


The actions taken and services 
used to reach specific outcomes 


The results of the teaching/ 
learning process 


Figure 2. Input, process, and outcome measures. 

Aligning inputs, processes, and outcomes is 
not a simple exercise. By design, LCAPs require 
districts to identify 

• overall goals aligned to state priorities, 

• expected annual measurable outcomes 
related to goals, and 

• actions and services used to achieve 
outcomes (see Figure 3). 


INPUTS 


PROCESSES 


OUTCOMES 
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GOAL: 

Related State and/or Local Priorities: 

1 2 3 4 5 6 7 8 

COE only: 9_ 10_ 

; Local : Specify 

1 Identified Need : 

| Goal App 

ica ter ’ S c h°°l s: 1 

1 

i Applicable Pupil Subgroups: 

1 

LCAP Year 1 : xxxx-xx 

EXPECTED ANNUAL 
MEASUREABLE 
OUTCOMES: 



ACTION 

5/ SERVICES: 

Scope of 
Service 

Pupils to be served within identified scope of service 

Budgeted 

Expenditures 



ALL 


OR: 

Low Income pupils English Learners 

Foster Youth Redesignated fluent English proficient 

Other Subarouos:(Soecifv) 



.ALL 


OR: 

Low Income pupils English Learners 

Foster Youth Redesignated fluent English proficient 

Other Subaroups:(Specifv) 


Figure 3. Section 2 ofLCAPs. 


One challenge in understanding the relations 
between multiple measures is a tendency in 
some districts to identify multiple expected 
annual measurable outcomes and lump 
numerous actions and services together under 
one goal. This approach does not show how 
the measures relate to one another. Separating 
inputs from processes within the actions and 
services section is the first step toward putting 
the full power of multiple measures into action. 

Attaching common symbols (e.g., A, B, C) to 
aligned inputs, processes, and outcomes is 
one simple method to help districts present 
information in a way that allows stakeholders 
to detect a district's theory of action for 
achieving overarching goals. Newport Mesa 


Unified School District in Orange County 
provides an example in Goal 2 of its 201 5/2016 
LCAP, 7 shown in Figure 4. 

Newport Mesa attached six distinct categories 
of expected annual measurable outcomes to 
this goal. The fourth category — Course-Taking 
Behavior — is signified by the letter D in Figure 
4. The district linked specific actions and 
services to its expected annual measureable 
outcome, showing the inputs and processes 
the district will prioritize (see Figure 5). 

For instance, Newport Mesa district officials 
plan to institute a schoolwide credit recovery 
program, a process we suspect they will use 
to increase the percentage of students who 
complete the a-g subject requirements. 
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GOAL 2: 

College and Career Readiness: Prepare grades 6-12 students to succeed in college and careers. 

• Increase CAHSEE pass rates for unduplicated count students. An analysis of CAHSEE pass rate data indicat- 
ed that unduplicated students (combination of low-income, EL, and foster students) in 201 2-201 3 scored 1 3 
percentage points below all other students in ELA and 1 8 percentage points below in math. Data also indi- 
cated that increasing the pass rate is correlated with increased overall academic achievement, thus increas- 
ing college and career readiness. 

LCAP Year 1:2015-2016 

Expected 

Annual 

Measureable 

Outcomes: 

A. College Entrance Exams 

• SAT/ACT: By fall 2015 establish a baseline number of high school students who have taken the SAT and/or 
ACT one or more times each school year. 

• PSAT: By June 201 6 maintain at least the same participation of grades 8 and 1 0 students as the number of 
participants in 2014-2015. 

B. Advanced Coursework 

• AP/IB Enrollment: By June 2016 increase the number of comprehensive high school students who enroll 
in Advanced Placement/International Baccalaureate (AP/IB) courses by at least 6%, as measured by enroll- 
ment lists. 

• AP/IB Pass Rates: By June 2016 increase the number of students who meet the minimum passing score on 
the AP/IB tests by at least 5%, as measured by the California Department of Education (CDE). 

C. Innovative Measures 

• Seal of Biliteracy: By June 201 6 increase the number of students who receive the California State Seal of 
Biliteracy from 1 24 to at least 1 50, as measured by the CDE. 

D. Course-Taking Behavior 

• University of California (UC) and California State University (CSU) A-G Completion: By June 2016 
increase the percentage of graduating seniors who complete a-g subject requirements from 51 .8% to at 
least 55%, as measured by the CDE. 

• Career Course Pathways: By June 2016 increase the percentage of students completing career course 
pathways by at least 3%, as measured by pathway completion lists. 

E. Career Preparedness Assessments 

• Career Survey: By June 2016 all grade 6, 8, and 1 0 students will complete the Naviance (online career/col- 
lege planning tool) career survey, as measured by completion lists. 

F. College and Career Planning 

• Academic Plans: By June 2016 every student in grades 6-12 will create a secondary academic plan focused 
on college. 


Figure 4. Newport Mesa Unified School District's LCAP (page 29). 
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D. Course-Taking Behavior 

• Advanced Via Individual Determination (AVID) district coordinator 

• AVID dues and memberships 

• AVID consultant to train coordinator 

$4,000 

$17,425 

$6,300 

All students at all high 
schools 

D. Course-Taking Behavior 

• School-wide initiatives: High School Credit Recovery 3.3 FTE, Life Skills 1.0 

FTE, Music 0.91 FTE, Reading 2.4 FTE, Art 0.33 FTE, Health Assistant 0.5 FTE 

$850,000 

All low-income and foster 
students at all high schools 

D. Course-Taking Behavior 

• CCSS site support: Embedded coaches from innovate ED for trainer coaching 

• CCSS site support: Embedded coaches training extra duty 

• CCSS site support: Embedded coaches substitutes 

$67,500 

$15,000 

$20,000 

All students at all middle 
and high schools 


Figure 5. Newport Mesa Unified School District's LCAP (page 3 1). 


The example in Figure 6, produced by the 
authors of this policy brief, takes Newport 
Mesa's method a step further by separating 
out inputs and processes within the actions/ 
services section. Consider a district that has the 
goal of increasing the number of students who 
complete an advanced course pathway and 
earn an industry certification or college credit. 
The expected annual measureable outcomes 
are the percentages of students who 

(a) complete an integrated course pathway 
and earn an industry certification or 
college credit; 8 

(b) complete two Career and Technical 
Education (CTE) course pathways and 
earn at least one industry certification; 9 

(c) score sufficiently 10 on two or 
more Advanced Placement (AP) or 
International Baccalaureate (IB) exams; 
or 

(d) complete two dual enrollment courses 
and earn college credit. 


Many inputs and processes need to be 
established in order to reach the overall goal 
of increasing the percentage of advanced 
course pathway completers, some of which are 
described in Figure 6. It is important to keep in 
mind the percentage of students completing 
an advanced course pathway will not increase 
significantly in just one year. Establishing the 
requisite structures and systems is the first 
step to ensuring any sustainable increase in 
advanced course pathway completers. 

Local communities with a similar goal might 
have different needs that are not reflected 
in Figure 6, which is not an exhaustive list. 
Additionally, there are few actions and services 
that do not interact with, or influence, each 
other. 11 For instance, increasing the percentage 
of students who complete an advanced course 
pathway can have positive effects such as 
promoting a school climate that supports 
college and career readiness. On the other 
hand, investing in specific course pathways 
might take resources away from other 
priorities, such as training in the Common Core 
State Standards for teachers. 




Increasing the percentage of students who complete an advanced course pathway 
and earn an industry certification or college credit (or eligibility through qualifying 
exam scores) 

(1 ) completing a CTE course pathway and earning an industry 
certification, and 

(2) completing an AP or IB course and scoring 3 or higher on an 
AP exam or 4 or higher on an IB exam 

completing two CTE course pathways and earning at least one 
industry certification 

completing two or more AP/IB courses and scoring 3 or higher 
on two AP exams or 4 or higher on two IB exams 
completing two dual enrollment courses and earning 
college credit 

Related Expected Annual 
Measureable Outcome 

Process Measures 

Establish formal relationships with local businesses 

Establish formal relationships with Regional Occupational Centers and Programs 
(ROCPs) 

Establish formal relationships with local postsecondary institutions 
Explicitly focus instruction on students' Deeper Learning skills, specifically 
critical thinking, collaboration, communication, and learning how to learn 
Explicitly focus instruction on students key transitional skills and knowledge, 
specifically the cultural norms across various workplaces and two- and four- 
year postsecondary institutions 
Input Measures 

Create an internal governance body that sets policies for (1 ) awarding industry 
certifications, (2) recognizing CTE course pathways, and (3) aligning dual 
enrollment and high school course credits 
Increase the number of teachers trained to teach CTE courses 
Increase the number of teachers in AP training programs 
Establish an IB Diploma Programme 
Increase the number of teachers in IB training programs 
Create a direct reimbursement program for AP/IB exam fees for 
qualifying students 

Create policies for transporting students to and from postsecondary institutions 



[A] 

[B] 



[A] 

[B] 



[A] 

[D] 


[A] 

[B] 

[C] 

[D] 

[A] 

[B] 

[C] 

[D] 


[A] [B] [D] 


[A] 

[B] 

[A] 

[C] 

[A] 

[C] 

[A] 

[C] 

[A] 

[C] 

[A] 

[D] 


Figure 6: Linking inputs, processes, and outcomes to increase the 
percentage of students who complete an advanced course pathway. 
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RECOMMENDATION #2: Tell Your 
Past, Present, and Future Stories 

Following Recommendation #1 will improve 
coherency and transparency while producing 
information that stakeholders can use to reach 
overarching goals. However, districts still need 
to present understandable information on 
past and current performance relative to the 
expectations of local communities. 

The Matrix Approach 

We endorse the use of matrices to describe 
what districts do well and where they can 
improve. The matrix approach is a cornerstone 
in recent thinking about multiple measures. 
Using matrices for accountability stems 
from the recognition that no single number 
calculated from an index of measures can 
provide the type of detailed information 
necessary for stakeholders to improve 
student outcomes. Single number or letter- 
rating systems, such as the API or A-F report 
cards, have traditionally held value among 
people seeking a simple answer to a complex 
question: What makes schools good or bad? By 
contrast, a district can use a matrix to set levels 
for current performance (low, mid-range, high) 
and trend data (declining, static, improving ). u 
Instead of simply presenting a flat number 
such as 65% of students graduated on time, a 
matrix supplements this percentage with low, 
signifying that the district and its stakeholders 
are not satisfied with the current graduation 
rate and declining to show that the graduation 
rate fell from the prior year. In doing so, a 
district can identify opportunities to improve, 
which are lost in a single letter grade or 
indexed number. 


Combining a matrix with Recommendation #1 
will create a system that satisfies the regulatory 
requirements placed upon districts and also 
produce a coherent set of information that 
a variety of stakeholders can use. A matrix 
suits LCAP development and can become 
an interactive, real-time data dashboard for 
external and internal use. Also, a matrix can exist 
exclusively in an LCAP as it is presented here. 

We recommend developing a multiple-measure 
matrix with the following characteristics: 

• goals and measures aligned to a district's 
definition of college and career readiness; 

• explicit relations between input, process, 
and outcome measures; and 

• clear articulation of what a district does 
well alongside its opportunities for 
improvement (see Figure 7). 

Of the three characteristics, the latter two 
are the mechanics that create coherency 
and produce the conditions necessary for 
actionability. However, the first characteristic — 
aligning goals and measures to a district's 
definition of college and career readiness — 
might be the most important. Often, process 
measures are where districts actualize 
their theory of action for achieving student 
outcomes. The fourth process measure in Figure 
7 shows teachers, parents, and community 
members that this district's theory for increasing 
the percentage of advanced course pathway 
completers relies on cultivating students' Deeper 
Learning skills. 13 At the heart of Deeper Learning 
is the notion that engaged students apply the 
knowledge from each course to past and future 
courses in ways that allow them to build new 
knowledge and work toward their aspirations. 



Metric 


Target Score Performance Trend 


Increasing the percentage of students who complete an advanced course 
pathway and earn an industry certification or college credit (or eligibility 
through qualifying exam scores) 


EXPECTED ANNUAL MEASUREABLE OUTCOMES 


[A] 

[B] 

[C] 

[D] 


Integrated pathway: (1) completing a CTE course pathway and 
earning an industry certification and (2) completing an AP/IB course 
and scoring > 3 on an AP exam or > 4 on an IB exam 

Career pathway: completing two CTE course pathways and earning at 
least one industry certification 

University pathway: completing two or more AP/IB courses and 
scoring > 3 on two AP exams or > 4 on two IB exams 

Postsecondary pathway: completing two dual enrollment courses 
and earning college credit 


% completers 90% 24% 

> completers 30% 5% 

i completers 20% 7% 

> completers 20% 5% 

i completers 20% 7% 



PROCESSES 


[A] 

[B] 
[D] 

[A] 

[B] 

[A] 

[C] 

[A] 

[C] 

[A] 

[C] 

[A] 

[C] 

[A] 

[D] 


Create an internal governance body that sets policies for (1 ) awarding 
industry certifications, (2) recognizing CTE course pathways, and (3) 
aligning dual enrollment and high school course credits 

Increase the number of teachers trained to teach CTE courses 
Increase the number of teachers in AP training programs 
Establish an IB Diploma Programme 
Increase the number of teachers in IB training programs 


Policies set 

student-teacher 

ratio 

student-teacher 

ratio 

Yes/No 

student-teacher 

ratio 


Create a direct reimbursement program for AP/IB exam fees for 
qualifying students 

Create policies for transporting students to and from 
postsecondary institutions 

Figure 7: Example of matrix approach 


Yes/No 

Yes/No 


Static 


Static 



[A] 

[B] 

Establish formal relationships with local businesses 

student - 
relationship ratio 

40 

90 

Mid-range 

Improving 

[A] 

[B] 

Establish formal relationships with Regional Occupational Centers 
and Programs (ROCPs) 

Yes/No 

Yes 

Yes 

High 

N/A 

[A] 

[D] 

Establish formal relationships with local postsecondary institutions 

% of institutions 

within 50 miles 

75% 

50% 

Mid-range 

Improving 

[A] 

[B] 

[C] 

[D] 

Explicitly focus instruction on students' Deeper Learning skills, 
specifically critical thinking, collaboration, communication, and 
learning how to learn 

student-teacher 

ratio 

30 

50 

Mid-range 

Improving 

[A] 

[B] 

[C] 

[D] 

Explicitly focus instruction on students' key transitional skills and 
knowledge, specifically the cultural norms across various workplaces 
and two- and four- year postsecondary institutions 

student-teacher 

ratio 

30 

50 

Mid-range 

Improving 

INPUTS 


Improving 
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Successfully completing an advanced course 
pathway requires students to understand 
how past courses provide the foundation for 
future courses and how earning an industry 
certification or college credit can be used as 
currency in multiple postsecondary pathways. 

A matrix also presents an opportunity for 
a district to establish college and career 
readiness targets that reflect community- 
specific assets and needs. For instance, the 
outcome rows in Figure 7 show a district with 
a healthy balance of students who aspire 
to enter a CTE field and four-year and two- 
year postsecondary institutions, as well as 
recognizing that some students might not 
have made definitive postsecondary plans. This 
district recognizes the diversity of aspirations 
by placing relatively equal value on each of 
the advanced course pathways. Rather than 
writing off students who are unsure of their 
aspirations, this district pushes such students 
toward the integrated course pathway, which 
leaves open many postsecondary options. 
Furthermore, by setting the completion 
target of 90%, this district recognizes that 
at least 1 0% of its students might aspire to 
attend the military, join the Peace Corps, or 
have other postsecondary plans not captured 
by these advanced course pathways. If this 
district instead knew that most students, but 
not all, aspired to enter four-year universities 
immediately after high school, the targets 
under [A], [B], [C], and [D] might be distributed 
as 1 0%, 1 0%, 60%, and 1 0%. Ultimately, local 
communities have the option to incentivize 
certain advanced course pathways without 
privileging one over the other. 


Producing data for the Score column that 
correspond to the Performance and Trend 
ratings can involve many different types of 
calculations depending on the measure. In 
our example, the score for the overall goal is 
simply the aggregated percentage of students 
who completed at least one advanced course 
pathway. Scores for the expected annual 
measurable outcomes reflect the percentage 
of students within the graduation cohort 
who completed one specific advanced course 
pathway. Note that districts can choose to 
double-, triple-, or quadruple-count students 
who complete multiple pathways or choose to 
count students toward only one pathway. We 
counted students only once. 

Measuring inputs and processes is slightly 
more challenging. For the first process 
measure — establishing formal relationships 
with local businesses — a district could target a 
ratio of relationships based upon the number 
of students (e.g., 40 students per relationship). 
The same method could apply to the other 
two input and process measures relative 
to AP and IB teacher training (see Figure 7). 
Similarly, certain measures may be constrained 
by available resources. For instance, a district 
might set the target of establishing dual 
enrollment agreements with 75% (or three) 
of the four postsecondary institutions within 
50 miles. Finally, other measures, such as 
establishing an IB Diploma Programme could 
be a simple Yes or No. 



Performance metrics (e.g., low, mid-range, or 
high) should correspond to scoring thresholds 
set for each measure. In this example, we set 
the following thresholds: 

• Low: 0-25% of graduating cohort 
completed at least one advanced course 
pathway, 

• Mid-range: 26-75% of graduating 
cohort completed at least one advanced 
course pathway, and 

• High: 76-100% of graduating cohort 
completed at least one advanced course 
pathway. 

Trend scores simply correspond to the previous 
year's performance (except for measures 
with Yes/No scores). Districts can decline or 
improve performance, or remain the same 
(i.e., static). There is no uniform or correct 
method for setting targets, creating scoring 
rubrics, or establishing thresholds for multiple 
measures. At the very least, the contents of the 
matrix should reflect district and community 
expectations for schools and students. 

Beyond setting targets and establishing 
scoring rubrics, a matrix allows all 
stakeholders to identify clear opportunities for 
improvement. The district in this example is not 
faring well on the university pathway outcome. 
However, the matrix shows that there are clear 
inputs and processes the district can improve 
upon to boost the percentage of students 
completing AP/IB courses and earning college 
credit eligibility through qualifying exam 
scores. Providing AP training to more teachers, 
creating a direct reimbursement system for 
students who cannot afford to pay for exam 


fees, and focusing instruction on Deeper 
Learning skills are some examples of such 
inputs and processes. 

Achieving outcomes should not be expected 
until a strong foundation of inputs is 
established and effective processes are 
implemented. The district in this example 
has made significant progress on the input 
and process measures. However, since this 
district is on the front edge of work to increase 
advanced course pathway completers, the 
targeted outcomes have yet to be reached. 

This may suggest that this district is making 
progress as measured by what some call 
the trailing indicators, and that the desired 
outcomes, or leading indicators, will be met 
soon. If all the targets for the input and process 
measures have been met, yet the outcomes 
are not changing, districts should consider 
reevaluating their theory of action for reaching 
the overarching goal. 

The matrix approach reflects a shift in 
traditional thinking around accountability, 
represented most recently in California by 
the API. The theory underlying API-style 
accountability is that when faced with 
potential punitive action, districts and schools 
will use the results from end-of-the-year tests 
to target instructional improvements that 
lead to better test scores in the following year. 
California's new approach to accountability 
seeks to shift focus from improving narrowly 
defined outcomes to improving entire 
systems through multiple measures. The Local 
Control Funding Formula (LCFF) provides 
districts with flexibility in how state allocated 
funds are spent and distributes additional 
funds to districts with high proportions 
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of disadvantaged students. LCAPs are the 
accountability mechanism used to ensure 
that these resource investments are used 
to increase performance relative to specific 
goals aligned to state priorities. The challenge 
with this type of system is making sense out 
of all the information districts are required to 
address in LCAPs. The matrix is an approach 
that allows districts to catalog multiple 
measures in a way that gives stakeholders a 
more holistic picture about what schools are 
doing well and where they need to improve. At 
the same time, the matrix approach provides 
district and school leaders with actionable 
information by developing coherence between 
input, process, and outcome measures. 

RECOMMENDATION #3: Use 
Infographics and Narrative Descriptions 
to Tell Your Story 

Implementing Recommendations #1 and #2 
can help drive a district toward continuous 
improvement, but they do not in themselves 
help a district tell a compelling story. Some 
districts have developed methods for telling 
theirs. Huntington Beach Union High School 
District summarized its LCAP in an infographic 
(see Figure 8). 14 The district's second overall 
goal is increasing student achievement 
and career readiness. The third page of the 
infographic features arrows and images to 
help stakeholders interpret the 1 2 expected 
measureable outcomes that the district 
aligned to this goal. The fourth page presents 
the inputs and processes that will be used 
to reach these outcomes. This infographic 
increases stakeholder understanding and 


lacks the density of an LCAP, but it does not 
show alignment across inputs, processes, 
and outcomes and does not provide detailed 
information on past and current performance. 

A narrative description can also help tell the 
story of a district's LCAP. Again, Huntington 
Beach provides an example in the executive 
summary of its 201 5/2016 LCAP. 15 The 
summary provides accessible language 
to describe an overall goal for college and 
career readiness, plus the process used to 
engage stakeholders. A narrative also has 
the potential to describe why the district 
selected the goals it did and its policies and 
practices for achieving those goals. Combining 
an infographic, a narrative description, and 
the other adjustments we recommend in 
this policy brief can produce the actionable 
information necessary to improve student 
outcomes, provide a transparent picture of 
overall performance, and tell a comprehensive 
and comprehensible story. 

CHALLENGES 

Our recommendations make LCAPs less about 
a single judgment of school quality and more 
about producing actionable information. 
However, a systems approach to improvement 
grounded in the use of multiple measures 
carries its own set of challenges. An infographic 
is one thing; creating a multiple measure 
system of accountability that is accessible 
and usable by a wide range of stakeholders 
is another entirely. Significant up-front 
investment will be needed to link multiple 
databases across schools. This investment 
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GOAL 

#1 



Increase student 
achievement & 
career readiness 


STATE 

PRIORITIES 


A 
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SCHOOLS 





EXPECTED 2015-16 MEASURABLE OUTCOMES 


4 A-G| t * 

+3% •mfr +2% 
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t 

+ 1 % 


INCREASE GRADUATES COMPLETING 
A-G REQUIREMENTS 


INCREASE COLLEGE ENROLLMENT 
IMMEDIATELY AFTER HIGH SCHOOL 


GRADS ENROLL IN ADULT SCHOOL 
CERTIFICATION PROGRAMS 


INCREASE 1 1 /l 2TH GRADE STUDENTS 
ENROLLED IN 1+ CTE COURSE 



INCREASE GRADUATION RATES 





- 10 % 

* 


SBAC ELA/MATH ACHIEVEMENT 


INCREASE STUDENTS PASSING AP 
EXAMS WITH SCORE OF 3+ 


DECREASE 2016 GRADS MISSING A-G 
REQUIREMENTS IN 1 SUBJECT AREA 




I BS 

BIB 




let Baseline IB 


STUDENTS PARTICIPATE IN ELA/MATH EAP 


DECREASE PERCENT OF CREDIT 
DEFICIENT SENIORS IN JUNE 
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INCREASE STUDENTS TAKING ACT 



INCREASE STUDENTS TAKING SAT 
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EXPECTED 2015-16 ACTIONS 


(§) Goal #1 


Action / Service 


1 . 1 Protocol to monitor student progress & communicate with parents for early intervention 

1 .2 Increase number of A-G approved CTE courses 

1 .3 Increase CTE offerings & student participation 

1 .4 Provide multiple pathways for student success after high school, inform parents of options 

1 .5 Additional options for credit recovery 

1 .6 Students/counselors/guidance specialists together develop 4-year educational/career plan 

1 .7 Remove barriers/provide support for all students to take higher level courses 

1 .8 ELD program ensures sufficient rigor for all EL students to meet annual objectives 

1 .9 Expand literacy & numeracy in all areas 

1.10 Provide best first instruction for all students 

1.11 Identify & counsel students & parents with potential to succeed in rigorous courses 

1.12 Formative assessments to inform instruction & student learning 

1.13 Use of technology in all areas to enhance & accelerate learning 

1.14 Increase student/parent understanding of cyber safety & security, responding effectively to related issues 

1.15 Evaluate/update technology & infrastructure 

1.16 Maintain/update hardware/software/internet systems 

1.17 Increase use of personal devices on campus to enhance learning 

1.18 Expand paperless work flow 

1.19 Provide training on student/parent portal 

1 .20 Increase interpretation/translation services at school functions 

1 .21 Increase awareness of Parent University 

1 .22 DELAC will develop & revise education & service plan for EL 

1 .23 Use parent groups to enhance communication to all stakeholders 


Target 




- 

Students 

Students 


Figure 8. Huntington Beach Union High School District's 201 5/2016 infographic (third and fourth pages only). 
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can take many forms, such as providing 
professional development, hiring full- or part- 
time technical experts, or purchasing software 
packages. Ensuring stakeholders across the 
system can enter, manage, and use data will 
likely require a combination of investments. 

Another challenge is incentivizing and 
supporting educators in using information 
continuously for diagnosing and solving 
problems of practice. Doing so would place 
more demands on educators whose time is 
already stretched thin by other initiatives. 
Cultivating a culture of trust, belief, and 
knowledge around data is both an input 
and a process aligned to producing a system 
that uses data to improve student outcomes 
continuously. Training across systems needs to 
go beyond one-off professional development 
programs. Establishing weekly supports for 
teams of educators to discuss collaborative 
data use might be one step toward creating 
such a culture. The research community has 
provided detailed recommendations in these 
areas and success and failure stories from 
districts across the U.S . 16 

Another challenge is a lack of evidence on 
how educators interact with and use data 
to improve student outcomes, particularly 
with systems that include input and process 
measures . 17 Identifying early what works 
and what does not will be necessary for 
disseminating best practices — and eliminating 
counterproductive ones. Furthermore, data 
cannot be actionable unless they reflect 
what is being measured. Understanding the 
technical considerations in choosing multiple 
measures is a required step toward ensuring 
that data is created and used responsibly . 18 


CONCLUSION 

Understanding how districts and their 
stakeholders make sense of multiple measures 
and use data generated from LCAPs to 
improve the college and career readiness 
of students might determine the success or 
failure of California's revised accountability 
system. However, LCAPs can benefit districts 
and schools regardless of the accountability 
implications. LCAPs can become the vehicles 
to improve the college and career readiness 
of students by harnessing community 
assets and tapping into the expertise of 
educators through informative and useful 
data. Implementing the recommendations 
in this policy brief will bring districts 
closer to transforming the A in LCAPs from 
Accountability to Actionability. 
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